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Abstract 

We present a new method for rendering novel images of flexible 3D objects from a 
small number of example images in correspondence. The strength of the method is the 
ability to synthesize images whose viewing position is significantly far away from the 
viewing cone of the example images ("view extrapolation"), yet without ever modeling 
the 3D structure of the scene. The method relies on synthesizing a chain of "trilinear 
tensors" that governs the warping function from the example images to the novel image, 
together with a multi-dimensional interpolation function that synthesizes the non-rigid 
motions of the viewed object from the virtual camera position. We show that two 
closely spaced example images alone are sufficient in practice to synthesize a significant 
viewing cone, thus demonstrating the ability of representing an object by a relatively 
small number of model images — for the purpose of cheap and fast viewers that can 
run on standard hardware. 
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1 Introduction 

In this paper we develop a reprojection technique for synthesizing novel views of a 3D 
object, given a collection of 2D model images in correspondence. We also consider the 
synthesis of novel views from a single model image and develop a method to control non- 
rigid transformations of the viewed object, such as facial expressions in the case of faces. 

The most significant aspect of our approach is the ability to synthesize images that are 
far away from the viewing positions of the sample model images without ever computing 
explicitly any 3D information about the scene. This property provides a multi-image repre- 
sentation of the 3D object using a minimal number of images. In our experiments, for exam- 
ple, two closely spaced frontal images of a face are sufficient for generating photo-realistic 
images from viewpoints within a 60 degrees cone of visual angle - further extrapolation 
is possible but the image quality degrades. The immediate application of our results is to 
provide a very fast 3D viewing system based on a small number of images that can run on 
standard hardware. 

The notion of image-based rendering is gaining momentum in both the computer graph- 
ics and computer vision communities. The general idea is to achieve photo-realistic virtual 
images while avoiding the computational-intensive process of acquiring a 3D model followed 
by rendering. Instead, one seeks to use a number of model images of the 3D object or scene 
as a representation from which novel views can be synthesized directly by means of image 
warping. 

The forerunner of this approach is to create a panoramic image of a scene (mosaic) from 
overlapping images taken from a fixed location while varying the orientation of the camera. 
The mosaic is mapped to a virtual cylinder that allows the user to look continuously at all 
directions but not to move. This is the basis for the QuickTimeVR system [7]. 

The fixed position constraint can be relaxed by computing the optical flow between the 
example images and using it to interpolate between the cylinders constructed at different 
locations (cf. [4, 5, 6]) (originally proposed for views, not mosaics, but the principle is the 
same). However, interpolation may produce physically-invalid images. Seitz and Dyer [22] 
proposed a physically- valid view interpolation method. The method involves recovering the 
epipolar geometry between the two acquired images and having interpolation done along 
the rectified epipolar lines. 

Interpolation can also be performed directly on the plenoptic function [1] which repre- 
sents the amount of light emitted at each point in space as a function of direction. Levoy 
et al. [16] and Gortler et al. [11] interpolate between a dense set of several thousands of 
example images to reconstruct a reduced plenoptic function (under an occlusion-free world 
assumption). They considerably increase the number of example images to avoid computing 



optical flow between the model images. 

The major limitations of the aforementioned techniques is that a relatively large number 
of model images is required to represent an object. The alternative approach, along the lines 
of this paper, is to reduce the number of acquired (model) images by exploiting the 3D-from- 
2D geometry of the problem with the aid of corresponding points between the model images. 
Laveau and Faugeras [15] were the first to use the epipolar constraint for view synthesis, 
allowing them to extrapolate, as well as interpolate, between the example images. Epipolar 
constraints, however, are subject to singularities that arise under certain camera motions 
(like when the virtual camera center is collinear with the centers of the model cameras) and 
the relation between translational and rotational parameters of the virtual camera and the 
epipolar constraint is somewhat indirect and hence requires the specification of matching 
points [15]. The singular camera motions can be relaxed by using the depth map of the 
environment. McMillan and Bishop [19] use a full depth map (3D reconstruction of the 
camera motion and the environment) together with the epipolar constraint to provide a 
direct connection between the virtual camera motion and the reprojection engine. Depth 
maps are easily provided for synthetic environments, whereas for real scenes the process is 
fragile especially under small base-line situations that arise due to the requirement of dense 
correspondence between the model images/mosaics [12]. 

In this paper we propose a new view-synthesis method that makes use of the recent 
development of multi-linear matching constraints, known as trilinearities, that were first 
introduced in [23]. The trilinearities provide a general (not subject to singular camera 
configurations) warping function from model images to novel synthesized images governed 
directly by the camera parameters of the virtual camera. Therefore, we provide a true 
multi-image system for view synthesis that does not require a companion depth map, nor 
the full reconstruction of camera parameters among the model cameras, yet is general and 
robust. The strength of our method is demonstrated by the ability to work with closely 
spaced acquired images yet synthesize high-quality views at a significant extrapolation from 
the viewing angles of the acquired images. Furthermore, our method can be generalized 
to work with a single acquired model image and to allow non-rigid transformations by 
integrating multi-linear constraints and multi- dimensional interpolation. 

The main contributions of our work are: 

1. The introduction of the trilinear tensor as the warping function. 

2. The derivation of a tensorial operator which is the heart of the method. The operator 
generates a cascading set of tensors from two model views in correspondence and the 
parameters of the virtual camera motion. The tensorial operator does not require the 
estimation of the baseline between the acquired images (typically a fragile process) 



thereby enabling the model images to be closely spaced without hindering much the 
robustness of the synthesis process. 

3. The combination of the tensor with a learning method for the generation of virtual 
views of an object given only a single model image. 

4. The combination of rigid transformations using the tensor with non-rigid transforma- 
tions achieved with multi- dimensional view interpolation. 

On the experimental side, we have tested the proposed method on a variety of objects 
with a variety of cameras in real-world conditions where neither camera calibration (or 
even camera type) is available nor the lighting conditions are controlled. We demonstrated 
that correspondence is practical for closely spaced images, and that the synthesis method 
is sufficiently accurate and robust. 

2 View Synthesis in Tensor Space 

The view synthesis approach is based on the following paradigm. Three views satisfy certain 
matching constraints of a trilinear form, represented by a tensor. Thus, given two views 
in correspondence and a tensor, the corresponding third view can be generated uniquely 
by means of a warping function, as described below in more detail. We describe how to 
recover the tensor parameters and show a "driver" function that governs the change in 
tensor coefficients as a result of moving the virtual camera. 

2.1 The Trilinear Warping Function 

The trilinear tensor concatenates together the camera transformation matrices (camera lo- 
cations) across three views, as follows. Let P be a point in 3D projective space projecting 
onto p,p',p" in three views ip,ip',ip" respectively, represented by the two dimensional pro- 
jective space. The relationship between the 3D and the 2D spaces is represented by the 
3x4 matrices, [1,0], [A,v ! ] and [B,v"], i.e., 
p = [/,0]P,p' = [A,v']Panrfp" = [B,v"]P 

where A, B stand for the rotational component of camera motion (generally these are 2D 
homography matrices) and v' , v" stand for the translational component (generally these are 
the epipolar points). 

We may adopt the convention that p = (x, y, 1) T , p' = (x 1 , y', 1) T , p" = (x", y", 1) T and, 
thus, P = (x,y,l,p). The coordinates (x,y),(x'y'),(x",y") are matching points across the 
three images. 




Figure 1: Each of the four trilinear equations describes a matching between a point p in the first 
view, some line s 1 ^ passing through the matching point p' in the second view and some line line r p k 
passing through the matching point p" in the third view. In space, this constraint is an intersection 
between a ray and two planes. 



The trilinear tensor is an array of 27 entries: 

4 k = v' j b$-v" k ai. i, j, k =1,2,3 
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where the covariant-contravariant indexing notation is assumed (see Appendix A). The 
tensor a\ forms the set of coefficients of certain trilinear forms that vanish on any corre- 
sponding triplet p,p',p": 

p^rte" = (2) 

where s'j are any two lines (s 1 - and s 2 ) intersecting at p' , and r p k are any two lines intersecting 
at p" (see Fig. 1). 

Since each of the free indices p,,p is in the range 1,2, we have 4 trilinear equations 
which are unique up to linear combinations. If we choose the canonical form where s 
and r represent vertical and horizontal lines, then the four trilinear forms, referred to as 
trilinearities, are expanded as follows: 

x"afp l - x"x'afp l + x'afp 1 - afp 1 = 0, 
y W - y"x'(xfp* + x'cLfp* - aJV = 0, 
x"cxfp* - x"y'(xfp* + y'cxfp* - cxfp* = 0, 
y"afp l - y"y'afp l + y'afp 1 - afp 1 = 0. 

Since every corresponding triplet p,p',p" contributes four linearly independent equa- 
tions, then seven corresponding points across the three views uniquely determine (up to 



5 



























image z 




Image 3 




Image 1 








\ c,t 


( 








Image 4 

















J k 



Figure 2: We generate tensor yj , that relates images 1,2 with some novel image 4, from the 



previous tensor a] (dashed) and the virtual camera motion parameters (C, t) from image 3 to 
image 4. Tensor a\ relates images 1,2 and 3 and is computed only once at the pre-processing 



stage. Tensor y\ is computed every time the user specifies a new (C,t). We use tensor yj to 
render the novel image (image 4) from example images 1,2. 



scale) the tensor aj . These constraints first became prominent in [23] and the underlying 
theory has been studied intensively in [27, 13, 24, 10, 28, 14, 25]. 

One can readily see that given two views in full correspondence and the tensor (recovered 
using 7 matching points with a third view), the entire third view can be synthesized by 
means of forward warping. From each trilinearity we can simply extract either x" or y", 
thus for every matching pair p,p' we can obtain p" . We then copy at p" the appropriate 
brightness value, for example the average of the pixel values at p and p' in the two model 
images. This process is referred to as "reprojection" in the literature. There are alternative 
ways of performing reprojection, but if we would like to do it without recovering first a 3D 
model of the scene, the trilinear tensor generally provides the best results since it is free 
from singular configurations (see [2, 23, 26]). 

We have described so far the implementation of the reprojection paradigm via the 
trilinear equations. In other words, given two model views and a tensor, the third view 
is uniquely determined and can be synthesized by means of a warping function applied to 
the two model images. In image-based rendering we would like to obtain the tensor via 
user specification of the location of a virtual camera, rather than by the specification of (at 
least) seven matching points. This is described next. 
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2.2 The basic Tensorial Operator 

The basic tensorial operator describes how to modify (transform) a tensor so as to represent 
a new configuration of three cameras. We are particularly interested in the case where only 
one camera has changed its position and orientation. Thus, by repeated application of the 
operator on a seed tensor with a sequence of desired virtual camera positions (translation 
and orientation) we obtain a chain of warping functions (tensors) from the set of acquired 
images (from which the seed tensor was computed) to create the desired virtual views (see 
Fig. 2). 

Consider the tensor a\ of the views < 1,2,3 > (in that order), and assume the user 
wishes to apply an incremental change of position of the third image, i.e., rotate the third 
camera position by the 3x3 coordinate matrix C, and translate it by the 3x1 translation 
vector t — this motion would result to a novel view, call it view 4. Then the tensor 7^ of 
the views < 1,2,4 > is given by: 

7 f = v%c k b\) - (v" l cf + t*X = <£«? - tk <- (3) 

This is so because we use Eq. 1 where we replace the motion parameters v" k \b k from 
the first image to the third image, with (v" l cf + t k ),(c k b\) which depend on the motion 
parameters from the first image to the novel one. The matrix a\ representing the rotational 
component of camera motion between the two model views 1,2 can be represented in closed 
form as a function of the tensor a\ as described in [21]: 
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where flx^Y^Z are rotation angles and a 2 stands for (a 
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To summarize, Eq. 3 is a general formula for transforming the tensor based on an 
incremental camera motion of a fixed (third) camera. Therefore, starting from a "seed" 
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Figure 3: View synthesis is divided into two parts. The pre-processing stage, done only once and 
the actual rendering done for every image. 

tensor and a sequence of desired camera motions, the set of corresponding tensors can be 
generated and used to warp the acquired images onto the novel views. We next consider 
how to obtain the seed tensor that starts the process. 

2.3 The Seed Tensor of Two Views 

Given two acquired images we can construct a special tensor composed of the elements of 
the fundamental matrix [9] that can serve as a seed tensor that starts the chain of tensors, 
as follows. Consider a configuration of three views in which views 2,3 coincide, i.e., Eq. 1 
becomes: 
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where 7^ is the tensor of the image triplet < 1,2,2 >. It can be readily verified that 



Jk 



the elements of 7^ are composed of the fundamental matrix fij = e^iv a-, — fij, and the 
remaining (nine) elements vanish. It will not be shown here, but the rank of 7/ is 2 whereas 
the rank of the tensor of three distinct views is 4 — but otherwise all other properties remain 
and, in particular, 7^ can serve as the first tensor that starts the synthesis process described 
above. 



2.4 The View Synthesis Loop 

We are ready to describe the full synthesis loop starting from a pair of closely spaced 
acquired (model) images of a 3D object. The method is divided into two stages — a 
preprocessing stage, done only once, and the actual rendering done for every new frame 
(see Fig. 3). 



1. Preprocessing 

(a) Compute dense correspondence (optical flow) between the two model images. We 
use a coarse-to-fine implementation of [18] described in [3]. Interactive tools for 
improving correspondence [17] can be used as well, although in our experiments 
optical flow alone was sufficient. 

(b) Recover the fundamental matrix of the two model images from the correspon- 
dences. We use a robust estimator based on a Monte-Carlo technique described 
in [20]. 

(c) Construct the rank-2 tensor (Eq. 5) from the elements of the fundamental ma- 
trix. 

2. View Synthesis 

(a) Accept camera motion parameters (rotation and translation) from the second 
camera to its new position. 

(b) Apply Eq. 3 to compute the tensor of the two model images and the novel one. 

(c) Synthesize the novel view by (forward) warping the two model views using the 
tensor computed in the previous step. 

2.5 Experiments 

We conducted four experiments, two of them shown here in Figs. 4, 5 and 6 and the 
remaining two are shown in the companion Video (and web page). In the "statue" sequence 
(Fig. 4) a pair of images of an African statue was captured with an SGI Indy Cam. The 
image size was 260 X 480 pixels. 

In the "Shannon" sequence (Figs. 5 and 6) the images were taken at a studio but 
no camera calibration was performed. The image size was 620 X 764 pixels. Due to the 
quality of the images we were able to extrapolate to over 90 degrees from the viewing cone 
defined by the example images. This is shown in Fig. 6. As expected, the quality of the 
optical flow determines the quality of the rendered image, especially when going to large 
extrapolations. Occlusions also become a significant source of noise especially along the 
boundaries. However, in all cases degradation in image quality is graceful. 

3 Synthesis from a single model view 

In the previous section, we have discussed how to synthesize new images for different view- 
points, given two examples images. Suppose now that only one image of a specific object, 
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say p nov , is available. Does our reprojection method break down? This section sketches a 
solution to this question. The idea is to create an additional - virtual - example image of 
object p nov from just one real view of it. Once one obtains a second image of the object 
from a different viewpoint, one can use the reprojection algorithm of section 2.4 to gen- 
erate subsequent virtual images. To accomplish this task without using a parametric 3D 
model, one may start from a collection of example views of another similar object p which 
plays the role of a prototype for representing generic transformations of the object class 
that the two objects belong to. Faces form such a class of objects. In general, we want to 
generate from one 2D view Img KOT of a 3D object p nov other views, exploiting knowledge 
of views of other objects of the same class. This idea of generating "virtual" views of an 
object by using class-specific knowledge has been discussed before (see references in [5]). 
Suppose that we have two views Img re t and Img of the prototype. We take Img re t to 
appear in the same pose as Img no ^. Img is a slightly transformed (i.e., rotated) view of 
Img re j (see diagram in Fig. 7). We can then compute the optical flow S p between these 
two views. Moreover, since the prototype object p is assumed to be "similar" to object 
Pnov, we assume that we can find good correspondence S nov between Img re j and Img no ^. 
We subsequently generate the optical flow S p+nov between the view Img re t and a new view 
of the object p nov by the vector addition: 

*^p-\-nov — *^p T ^nov \® ) 

A new view, Img p+nov , of object p nov is then rendered by texture mapping from the 
single available view Img no ^ after forward warping from Img re j using optical flow S p+nov . 
In a sense, we "map" the learned transformation (optical flow S p ) from Img re j to Img no ^ 
using flow S nov . We now have two images of object p nov that our reprojection technique 
can use to simulate a virtual camera and generate new images and image sequences. 

We demonstrate this technique in Fig. 7 using as an example a self-portrait of Van 
Gogh. A slight rotation is learned from another similar prototypical "object", in this case 
another face, to generate a first virtual image of Van Gogh. Then the reprojection method 
of section 2.4 is used to generate subsequent views. 

4 Incorporating Non-rigid Transformations 

So far we have described a technique that allows the user to generate new images by 
controlling the rigid degrees of freedom that correspond to motion of the camera. From two 
or more images in correspondence it may also be possible to generate new images of non-rigid 
3D objects as a function of input parameters that correspond to non-rigid transformations 
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such as a change in expression of a face. The underlying operation is multidimensional 
interpolation, a simple extension of traditional image morphing. We outline the technique 
and illustrate how it can be integrated with the algebraic method described so far. 

Let us assume that n images are available and that pixel-wise correspondence can be 
computed with an optical flow algorithm between one of them, chosen as the "zero" reference 
image, and each of the n — 1 others. As we saw earlier, correspondence associates to each 
image i the optical flow, that we note as S 8 -, of the position of each pixel relative to the 
reference image. We can also associate to each image a vector of color values, the "texture" 
vector, that we note as TV The texture vector T\- is simply the image i warped to the shape 
of the reference image by the optical flow S 8 -. Let us also assume that the user defines the 
values Yi of the non-rigid parameters of interest to be associated with each one of these 
"example" images. 

A multidimensional interpolation technique such as Radial Basis Functions or splines is 
then used to interpolate the n example pairs (r 8 -, (S 8 -, T 8 )) (see for instance [5]). The mapping 
from the input space of non-rigid parameters to the output space of images, expressed in 
terms of textures and flows, is provided by the following interpolation scheme which can be 
regarded as a learning network ([4, 5]) 

S(r) = £?=iC,-G(r-r,-), 

(7) 
T(r) = £? =1 a,-G(r-r,-), 

where the c 8 - and a 8 are vectors of coefficients, and G is a basis function, which may be 
a radial basis function, like the Gaussian or a spline, like a tensor product spline. The 
network coefficients c 8 - and a 8 are found by solving the linear system of equations (7) over 
the training data ([4, 5]). 

Given a new vector r of non-rigid parameters, the network of Eq. 7 synthesizes a new 
(S,T) using the learned coefficients c 8 - and a,-, which is then rendered in a new image 
by warping T\- according to the warping field S 8 -, effectively performing multidimensional 
morphing. 

This simple technique can be used to control several non-rigid degrees of freedom such as 
facial expressions, as shown by [4, 5, 17, 8]. It can be combined directly with the algebraic 
technique described earlier to control the position of the virtual camera. Thus, given a set 
of images with different non-rigid parameters from different viewpoints (for each viewpoint 
we have an image for each of the non-rigid parameters), the algorithm is composed of the 
following steps: 

Preprocessing: 
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1. For each of the viewpoints (rigid parameters) choose a reference image and compute 
dense correspondence (optical flow) between this reference and the other images repre- 
senting the non-rigid parameters. Obtain the texture vectors for each of these images 
- relative to the corresponding reference image. 

2. For each of the viewpoints use the example images to learn the mapping from the 
non-rigid parameters to the flow and texture vectors found in the previous step. 

3. For one of the non-rigid parameters (any one will do) follow the preprocessing steps 
of the algorithm described in section 2.4. 

View Synthesis: 

1. Accept the values of the rigid and non-rigid parameters. 

2. For the desired set of values of the non-rigid parameters use the multidimensional 
interpolation algorithm to generate a virtual image for each of the viewpoints. 

3. Apply the reprojection algorithm described in section 2.4 using the accepted values 
of the rigid parameters and the images generated in step 2 to obtain an image from 
the desired viewpoint. 

Fig. 8 shows the simple case of one non-rigid degree of freedom. Given four images, cor- 
responding to two values of the non-rigid parameter and two viewpoints, a virtual image for 
the desired intermediate expression is obtained by interpolation for each of the viewpoints. 
Then our reprojection technique generates views of the intermediate expression from the 
desired new viewpoint. 

5 Conclusions 

The method we describe in this paper can render novel images of flexible 3D objects from 
a small number of example images without the need of an explicit 3D model. Its main 
strength is the ability to synthesize images whose viewing position is significantly far away 
from the viewing cone of the example images. 

Clearly the key step in this class of techniques is the computation of pixel-wise corre- 
spondence between the example images. We addressed this problem by using an optical 
flow algorithm from the computer vision literature that estimates dense sets of pixel-level 
correspondences. It is well known that correspondence is a very difficult problem which can 
be solved only for images that are similar enough and do not suffer from significant self- 
occlusions. 3D model-based approaches, however, suffer from even worse correspondence 
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problems, if the 3D models are themselves estimated from a set of images. Our technique 
has a major advantage relative to others, at least for the rigid degrees of freedom of the 
camera, since it relies on pairs of images with a small baseline, which helps the critical 
correspondence stage. Methods that estimate 3D structure are very noisy with small base- 
lines. Morphing techniques, such as [22], require large baselines since they cannot perform 
extrapolation. 

The problems with correspondence and occlusions can be solved at the expense of in- 
creasing the number of examples. An apparently deeper problem is how does the approach 
to synthesis scale with the number of pose/expression parameters? It would seem that 
increasing the number of parameters used to control image synthesis may require an expo- 
nential increase in the number of example images. Fortunately, there are several reasons 
why the problem is not as bad as it may seem. First, our technique needs only two images 
from close viewpoints to deal with all the rigid degrees of freedom. Second, the number of 
examples required for the non-rigid degrees of freedom may remain quite low, for appro- 
priate choices of the input parameters (see [5] and references therein). Third, it is possible 
to bypass the curse of dimensionality problem by representing an object as a hierarchy of 
components. Interpolation networks responsible for each component are thus independent, 
for instance the mouth separately from the eyes [17, 8]. The main factors likely to set a 
lower bound on the number of example images needed for a given set of control parameters 
are therefore simple visibility constraints: every part of the scene to be rendered must be 
visible in at least two of the example images. 
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A On Tensorial Notations 

We use the covariant-contravariant summation convention: a point is an object whose coor- 
dinates are specified with superscripts, i.e., p l = (p 1 , p 2 , ...). These are called contravariant 
vectors. An element in the dual space (representing hyper-planes — lines in V 2 ), is called 
a covariant vector and is represented by subscripts, i.e., Sj = (si, S2, ••••)• Indices repeated 
in covariant and contravariant forms are summed over, i.e., p 8 s 8 - = p 1 si + p 2 S2 + ... + p n s n . 
This is known as a contraction. For example, if p is a point incident to a line s in V 2 , then 
p l Si = 0. Vectors are also called 1-valence tensors. 2-valence tensors (matrices) have two 
indices and the transformation they represent depends on the covariant-contravariant posi- 
tioning of the indices. For example, a\ is a mapping from points to points, and hyper-planes 
to hyper-planes, because a\p % = q J and ajsj = r 8 - (in matrix form: Ap = q and A 1 ' s = r); 
aij maps points to hyper-planes; and a l ' J maps hyper-planes to points. When viewed as a 
matrix the row and column positions are determined accordingly: in a\ and aji the index i 
runs over the columns and j runs over the rows, thus b-a] = c- is BA = C in matrix form. 
An outer-product of two 1-valence tensors (vectors), a 8 -6 J ', is a 2-valence tensor c\ whose i,j 
entries are aft 3 — note that in matrix form C = ba T . The tensor of vector products is de- 
noted by eijk (indices range 1-3) operates on two contravariant vectors of the 2D projective 
plane and produces a covariant vector in the dual space (a line): ejjkp'q 3 = Sk, which in 
vector form is s = p X q, i.e., s is the vector product of the points p and q. 
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